<html>
<head><meta charset="utf-8"><title>cloning all the submodules? · t-compiler/help · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/index.html">t-compiler/help</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html">cloning all the submodules?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="199897242"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199897242" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199897242">(Jun 05 2020 at 15:57)</a>:</h4>
<p>Is there a reason that everyone should have to clone all of the submodules when building? It takes a rather long time and wastes space, but I think it is only needed for building a distribution right?</p>



<a name="199897651"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199897651" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199897651">(Jun 05 2020 at 16:00)</a>:</h4>
<p>kinda, cargo's lockfile resolution requires that you have Cargo.toml's around at least</p>



<a name="199897665"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199897665" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199897665">(Jun 05 2020 at 16:00)</a>:</h4>
<p>(for the crates in the workspace anyway)</p>



<a name="199897721"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199897721" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199897721">(Jun 05 2020 at 16:01)</a>:</h4>
<p>I <em>do</em> think that improvements here can and should be made, but it's not obvious what the right workflow is for some things</p>



<a name="199897773"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199897773" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199897773">(Jun 05 2020 at 16:01)</a>:</h4>
<p>I have wanted to just entirely not have submodules in the past, and instead have a list of SHAs and repositories in some file</p>



<a name="199897844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199897844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199897844">(Jun 05 2020 at 16:01)</a>:</h4>
<p>personally I think a good 70-80% of the problems with submodules are just because of the UI/UX that git imposes</p>



<a name="199902436"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199902436" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199902436">(Jun 05 2020 at 16:33)</a>:</h4>
<p>But why are they needed at all? Or at least, why can't we just list them in <code>Cargo.toml</code> as <code>foo = { git = "something on github" }</code>?</p>



<a name="199902619"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199902619" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199902619">(Jun 05 2020 at 16:35)</a>:</h4>
<p>Doing so would make it harder to change both rustc and a submodule at the same time. Also it will cause Cargo to clone the repo anyway.</p>



<a name="199902857"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199902857" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199902857">(Jun 05 2020 at 16:36)</a>:</h4>
<p>Oh, hmm... that's true</p>



<a name="199902988"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199902988" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199902988">(Jun 05 2020 at 16:37)</a>:</h4>
<p>But it still seems odd, e.g. that I need to clone rust-by-example to build the compiler... or the book or the rustc-dev-guide or llvm if I'm using system llvm</p>



<a name="199903133"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199903133" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199903133">(Jun 05 2020 at 16:38)</a>:</h4>
<p>Would it be hard to make these not cloned by default?</p>



<a name="199903243"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199903243" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199903243">(Jun 05 2020 at 16:39)</a>:</h4>
<p>For llvm <code>x.py</code> could skip the submodule update if you set <code>config.toml</code> to use the system LLVM. I believe it has to parse <code>config.toml</code> anyway.</p>



<a name="199905892"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199905892" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199905892">(Jun 05 2020 at 17:00)</a>:</h4>
<p>Also, I just noticed that one can set <code>submodules = false</code> in <code>config.toml</code>!</p>



<a name="199905917"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199905917" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199905917">(Jun 05 2020 at 17:01)</a>:</h4>
<p>Is there any reason we shouldn't recommend doing this?</p>



<a name="199905997"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199905997" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199905997">(Jun 05 2020 at 17:01)</a>:</h4>
<p><span class="user-mention silent" data-user-id="133247">bjorn3</span> <a href="#narrow/stream/182449-t-compiler.2Fhelp/topic/cloning.20all.20the.20submodules.3F/near/199903243">said</a>:</p>
<blockquote>
<p>For llvm <code>x.py</code> could skip the submodule update if you set <code>config.toml</code> to use the system LLVM. I believe it has to parse <code>config.toml</code> anyway.</p>
</blockquote>
<p>Looking through the code, it appears this is already the case</p>



<a name="199906113"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199906113" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199906113">(Jun 05 2020 at 17:02)</a>:</h4>
<p>I think that would be ok if someone just wants to work on std or something but if they ever decide to do something with the compiler, they're probably going to run into weird errors</p>



<a name="199906173"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199906173" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199906173">(Jun 05 2020 at 17:02)</a>:</h4>
<p>Which can be frustrating as a newcomer</p>



<a name="199906466"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199906466" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199906466">(Jun 05 2020 at 17:05)</a>:</h4>
<p>In some respects, I think it's better to have a long-ish process that consistently works than to have a short-cut process that doesn't always work. You can start that long process and then go to bed or something while it works.</p>



<a name="199906550"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199906550" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199906550">(Jun 05 2020 at 17:06)</a>:</h4>
<p>I wonder if it would be worth while to investigate using shallow clones for some or all of those submodules though...</p>



<a name="199907048"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199907048" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199907048">(Jun 05 2020 at 17:11)</a>:</h4>
<p>I'm actually kind of conflicted here... on one hand I agree that having something that consistently work is best, but on the other hand long build times is one of the top things that people have complained about in the survey</p>



<a name="199907187"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199907187" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199907187">(Jun 05 2020 at 17:12)</a>:</h4>
<p>moreover, as I've been writing the "Getting started" PR, I'm finding that most of our recommended ways of doing things are "probably going to work, but could lead to weird issues"</p>



<a name="199907643"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199907643" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199907643">(Jun 05 2020 at 17:16)</a>:</h4>
<p>Yeah, that's totally fair! I don't want to stand in the way of progress, just providing a different perspective <span aria-label="slight smile" class="emoji emoji-1f642" role="img" title="slight smile">:slight_smile:</span></p>



<a name="199920769"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199920769" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199920769">(Jun 05 2020 at 19:04)</a>:</h4>
<p>Not at all :)</p>



<a name="199921692"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199921692" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199921692">(Jun 05 2020 at 19:13)</a>:</h4>
<p>I think the major difficulty is that the only really correct way to compile the compiler is _really_ slow, and so we have all of these hacky shortcuts that usually work, but sometimes not... and that's what we actually have to teach people</p>



<a name="199927669"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199927669" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199927669">(Jun 05 2020 at 20:06)</a>:</h4>
<p>I think getting a document put together which lays out the "right way" to do a bunch of workflows would be a great start, because right now it's really hard (for me at least) to even know what to optimize</p>



<a name="199927704"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199927704" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199927704">(Jun 05 2020 at 20:06)</a>:</h4>
<p>like I know <em>I</em> have things that sometimes annoy me, but I doubt that they're the primary concern of most newcomers</p>



<a name="199927755"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/cloning%20all%20the%20submodules%3F/near/199927755" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/cloning.20all.20the.20submodules.3F.html#199927755">(Jun 05 2020 at 20:07)</a>:</h4>
<p>and when we do update things it'd be much better if we could get the word out by updating a central document and pinging compiler team rather than very slowly that knowledge percolating through the compiler team and eventually reaching new contributors</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>